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THE  PERCEPTRON  AS  AN  ADAPTIVE  CLASSIFICATION  DEVICE 

Prepared  by: 

C.  Nicholas  Pryor,  Jr. 


ABSTRACT:  Many  data  reduction  problems  fall  under  the  general 
category  of  classification,  such  as  sonar  target  classification 
or  character  recognition.  These  may  be  represented  as  a  process 
of  mapping  a  hyperspace  representing  the  Input  data  Into  an  out> 
put  space  representing  distinct  decisions  or  categories.  The 
majority  of  practical  problems  may  be  simplified  by  the  assump¬ 
tion  of  continuity  In  the  mapping  process,  and  It  Is  desirable 
to  mechanize  the  classification  problem  according  to  this 
assumption. 


A  class  of  adaptive  mechanisms  known  as  learning 
maohlzias  has  the  capability  of  classifying  input  data,  and  a 
machine  of  this  sort  can  adjust  Itself  to  satisfy  the  desired 
classifying  criterion  if  it  is  given  a  collection  of  Input  data 
identified  as  to  its  desired  class.  One  machine  of  this  type  is 
known  as  the  Peroeptron  and  is  discussed  in  some  detail.  A 
simpler  form  of  the  maobine,  the  half -Peroeptron,  is  also  dls- 
ouased* 


These  two  types  of  learning  machines  were  simulated 
on  the  IBK  704  ooBqi>uter,  and  sample  problems  in  classlf loatlon 
were  processed  on  them.  The  results  demonstrate  some  of  the 
capabilities  and  limitations  of  the  two  machines. 

A  discussion  Is  also  included  on  the  possible  future 
role  of  learning  machines,  and  on  their  application  to  naval 
problems . 
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This  report  gives  the  results  of  prograxnmlng  an  IBM  704 
Computer  to  act  as  a  PERCEPTRON  (a  self-adaptive  or  learning 
machine)  on  two  simplified  classification  problems  as  a 
preliminary  to  a  study  of  submarine  classification  with  sonar 
data.  The  work  on  this  project  was  supported  under 
Task  No.  RI7SD-4C150»  PUl^S  Technical  Direction.  The  report  . 
Is  for  the  information  of  other  scientists  Interested  in 
classification  pTOblems  or  in  the  use  of  computers  as 
adaptive  machines. 

W.  D.  COLEMAN 
Captain,  TJSN 
Commander 
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THE  PERCEPTROH  AS  AH  ADAPTIVE  CLASS  IP  ICATIOH  DEVICE 


THE  CLASSIFICATION  PROBLEM 

1.1  In  many  problems  in  data  reduction  such  as  character 
recognition  and  sonar  target  classification,  a  large  amount  of 
data  is  available  (such  as  light  intensity  at  many  points  on  a 
retina  or  spectral  intensity  at  many  frequencies);  and  the  prob¬ 
lem  is  to  Identify  a  given  set  of  input  data  as  belonging  to  one 
of  several  possible  disjoint  classes  or  categories  of  input  sig¬ 
nal,  each  of  which  must  give  rise  to  a  distinct  decision  or  out¬ 
put  state  of  the  data  reduction  system.  If  there  are  N  pieces 
of  input  data,  each  with  a  number  of  discrete  or  a  range  of  con¬ 
tinuous  values,  the  input  to  the  system  may  be  represented  by  a 
point  in  an  N-dlmenslonal  ’’input  space”.  Correspondingly  if 
there  are  M  possible  output  decisions  that  may  be  made  (that  is, 

M  possible  output  classes),  we  may  think  of  the  output  as  a  one- 
dimensional  space  containing  M  discrete  points;  and  the  problem 
becomes  one  of  a  many-to-one  transformation  of  points  from  the 
input  space  to  the  output  apace.  Another  way  of  considering  the 
problem  is  that  of  labeling  every  point  in  the  input  space  with 
the  identification  (perhaps  a  number  from  1  to  M)  of  the  output 
state  appropriate  to  it. 

1.2  So  far  we  have  assumed  that  the  output  state  required 
for  a  given  point  in  input  space  is  Independent  of  that  required 
for  any  other  point,  and  that  it  is  necessary  to  examine  each 
point  in  detail  to  deteralne  its  required  output.  This  is  an 
overwhelming  Job  in  any  reasonable  problem  and  of  course  impossi¬ 
ble  if  any  one  of  the  input  variables  is  continuous.  However, 
this  is  the  approach  which  must  be  taken  if  any  of  the  usual 
switching  circuit  or  logical  coxqputer  realisations  is  to  be 
attempted. 

1.3  In  most  problems  of  practical  Interest  it  is  reasonable 
to  assume  that  small  changes  in  one  or  more  of  the  input  variables 
will  tend  not  to  change  the  desired  output  state.  That  is,  near¬ 
by  points  in  the  Input  space  will  tend  to  belong  to  the  same 
class,  and  the  transformation  from  input  space  to  output  space 
will  tend  to  be  continuous.  The  problem  now  becomes  a  simpler 
one  of  determining  the  boundaries  separating  regions  of  input 
space  belonging  to  different  classes.  It  is  no  longer  necessary 
to  study  every  possible  input  state,  but  only  enough  to  estimate 
these  boundaries  in  the  input  space.  New  points  (that  is  points 
which  were  not  Initially  used  in  establishing  the  boundaries) 

az^  then  categorised  according  to  the  region  in  which  they  fall. 
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which  means  that  they  are  assumed  to  belong  to  the  same  class 
as  neighboring  points.  If  this  decision  proves  to  be  wrong,  it 
is  then  necessary  to  adjust  the  boundaries  or  perhaps  to  form  a 
new  region  to  take  the  new  point  into  account. 

l.k  In  spite  of  this  simplification,  determining  the 
proper  boundaries  from  a  collection  of  input  data  and  desired 
outputs  is  still  an  enormous  Job  (made  even  more  so  by  the 
difficulty  of  thinking  In  more  than  three  dimensions)  and  it 
becomes  desirable  to  mechanize  this  operation.  In  general  this 
problem  is  handled  by  some  sort  of  adaptive  mechanism  to  which 
a  sample  of  input  data  is  fed,  along  with  information  concerning 
the  desired  output.  Internally,  the  machine  consists  of  a  large 
logical  network,  whose  logical  function  may  be  varied  by  adjust¬ 
ment  of  some  parameters  of  the  network.  Certain  types  of  logi¬ 
cal  systems  have  the  desired  characteristic  of  tending  to  pro¬ 
duce  the  same  output  for  similar  Inputs,  and  these  networks  may 
be  thought  of  as  defining  the  regions  of  input  space  for  which 
a  given  output  will  occur.  If  the  machine  output  differs  from 
the  desired  output,  a  corrective  signal  is  fed  back  to  the 
machine.  Ihla  performance  feedback  is  designed  to  adjust  the 
internal  construction  of  the  logical  network  in  such  a  way  as 
to  correct,  or  reduce  the  probability  of,  the  wrong  answer. 

The  effect  of  this,  in  a  properly  designed  mechanism,  is  to 
move  the  boundaries  between  regions  of  input  space  in  Shuch  a 
way  that  the  given  input  point  will  be  included  in  the  proper 
region. 

1.5  In  an  Ideally  designed  machine,  only  the  boundary  in 
question  will  be  moved;  and  training  for  one  input  point  will 
not  affect  the  response  for  other  points  outside  its  immediate 
vicinity.  It  is  usually  not  possible  to  attain  this  degree  of 
independence  in  a  practical  mechanism,  and  all  boundaries  will 
tend  to  move  during  a  given  training  operation.  This  reduces 
the  convergence  to  the  proper  configuration  to  rin  iterative 
procedure  of  successive  approximationa,  and  it  is  usually  neces- 
sazT’  to  go  through  a  given  finite  list  of  input  data  a  number 
of  times  before  error-free  operation  can  be  obtained.  This 
gives  the  machine  a  statistical  appearance,  even  though  the  in¬ 
ternal  operation  is  usually  not  randomly  determined,  and  the  . 
size  of  the  machine  generally  makes  anything  other  than  statist! 
cal  analysis  Impraotioal.  For  this  reason,  it  is  usually  suf¬ 
ficient  in  the  training  procedure  simply  to  make  corrections  in 
the  direction  of  including  the  input  point  in  the  desired 
region  rather  than  necessarily  forcing  the  machine  to  produce 
the  desired  answer  before  going  on  to  the  next  input  sample. 

1.6  The  complexity  of  the  Internal  logical  network  of  the 
adaptive  system  determines  the  degree  of  detail  in  the  appor¬ 
tioning  of  iiqput  space  which  may  be  stored  in  the  memory  of  the 
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logical  network.  Where  the  Input  apace  la  divided  Into  many 
•eparate  reglona  with  complicated  boundarlea,  a  very  elaborate 
switching  circuit  would  be  required  to  realize  the  logical 
functions  Implied  by  these  regions.  Thus  It  would  be  expected 
that  a  rather  larg^  adaptive  logical  network  might  be  required 
to  simulate  this  operation.  Conversely,  certain  simple  geome¬ 
tries  of  the  regions  In  It^ut  space  may  be  realized  using  rather 
small  logical  networks.  In  addition  to  this,  many  of  the  simple 
logical  systems  which  may  be  used  In  an  adaptive  learning 
machine  have  fundamental  limitations  which  prevent  them  from 
being  able  to  realize  certain  geometries  within  the  Input  space. 
These  weaknesses  generally  become  apparent  upon  careful  study  of 
the  philosophy  used  In  the  machine  organization. 

1.7  The  following  sections  of  this  report  discuss  the 
organization  of  two  types  of  adaptive  classification  or  learning 
machines.  Their  simulation  on  an  IBM  704  computer  Is  also  dis¬ 
cussed,  and  sample  problems  are  shown  which  demonstrate  the 
oapabllltles  and  characteristics  of  the  machines  further. 


THE  PERCEPTRON 

2.1  A  device  which  seems  rather  well  suited  for  the  classi¬ 
fication  problem  Is  the  Perceptron,  developed  at  the  Cornell 
Aeronautical  Laboratory.  The  Perceptron  was  developed  as  a  model 
of  the  possible  construction  of  the  human  nervous  system,  and 
thus  Its  components  may  be  Identified  with  the  receptor  neurons, 
motor  neurons,  and  Intermediate  "gray  matter"  neurons  In  the 
nervous  system.  The  organization  of  the  device  is  shown  In 
Figure  1.  Inputs  to  the  system  are  represented  by  the  states  of 
the  Stimulus  (3)  Units,  and  through  the  logical  network  the 
excitations  to  the  Response  (R)  Units  are  determined.  The  binary 
state  of  the  several  R  Units  Is  considered  to  be  the  output  of 
the  device. 


2.2  Th  logical  function  is  carried  out  by  the  intermediate 
Association  t  Units,  which  act  as  the  connections  between  the 
S  and  R  Units.  Each  A  Unit  has  a  number  of  Input  connections 
(dendrites)  which  are  randomly  connected  to  several  of  the  S 
Units.  The  output  of  the  A  Unit  Is  connected  at  random  to  an 
Input  of  one  of  the  R  Units.  Of  course,  no  generality  Is  lost 
by  reordering  the  A  Units  so  that  all  those  connected  to  a  given 
R  Unit  are  located  together  as  shown  In  Figure  1.  Bach  A  Unit 
Is  a  two-state  device  which  Is  either  on  or  off  depending  on 
whether  the  total  excitation  to  the  A  Unit  (sum  of  all  Its  In¬ 
puts)  exceeds  a  given  threshold  level.  If  the  A  Unit  Is  on,  a 
signal  proportional  to  Its  "Value"  Is  placed  on  its  output 
temlnal.  If  It  Is  off,  either  no  signal  or  minus  the  "Value" 
is  placed  on  the  output,  depending  on  the  construction  of  the 
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particular  machine.  Each  R  Unit  then  becomes  either  on  or  off 
depending  on  whether  Its  Input  excitation  (the  sum  of  the  dut-p 
puts  of  all  A  Units  connected  to  It)  Is  above  or  below  a  fixed 
threshold. 

2.3  In  the  nervous  system  the  output  of  each  neuron  Is  a 
signal  of  fixed  magnitude,  and  the  triggering  level  or  threshold 
of  the  neuron  varies  according  to  some  electrochemical  process 
within  the  neuron.  Learning,  or  modification  of  the  logical 
network  within  the  nervous  system,  Is  accomplished  by  varying 
the  triggering  level  of  the  Individual  neurons  according  to  the, 
reinforcement  given  to  certain  responses  and  to  whether  the 
neurons  were  active  during  the  response.  In  the  Perceptron,  the 
thresholds  of  the  Individual  A  Units  are  kept  fixed,  but  their 
Values  are  varied  according  to  the  reinforcement  given  the 
qiaehlne.  The  two  methods  of  learning  az*e  roughly  equivalent, 
but  In  the  Perceptron  with  Its  smaller  number  of  components 

(roughly  10^  A  Units  against  10^^  neurons  In  the  brain)  finer 
detail  of  learning  may  be  accomplished  by  varying  the  Values 
and  keeping  the  thresholds  fixed.  Of  course  It  Is  necessary  to 
adjust  the  Values  during  the  training  operation  In  some  system¬ 
atic  way  that  will  tend  to  converge  toward  solution  of  the 
problem.  A  number  of  procedures  for  adjusting  the  A  Unit  Values 
have  been  studied  and  their  characteristics  reported  by  Cornell 
Aeronautical  Laboratory. 

2.4  Each  of  the  A  Units  may  be  thought  of  as  a  basic 

logical  element  which  performs  a  boolean  operation  on  Its  Inputs. 
In  particular  this  operation  Is  the  symmetric  function  Sm  „ 

where  there  are  N  Inputs  to  the  A  Unit  and  the  threshold  Is  T. 

The  li^uts  to  the  A  Unit  may  be  restricted  to  the  outputs  of  the 
S  Units,  or  for  a  more  general  logic  both  the  S  Unit  outputs 
and  their  complements  may  be  made  available  as  Inputs  for  the  A 
Units.  The  excitation  to  the  R  Units  Is  then  some  linear  combi¬ 
nation  of  the  symmetric  functions  realized  by  the  A  Units,  and 
It  is  this  linear  combination  that  is  varied  during  the  learning 
process.  Finally  the  greater-than-  or  less-than  decision  In  the 
R  Unit  produces  the  last  stage  of  logic.  Thus  the  overall 
machine  looks  like  a  logical  network,  althou^  It  Is  really  com¬ 
posed  of  two.  nonlinear  elements  separated  by  a  linear  operation. 
In  general  the  number  of  possible  logical  functions  which  the 
machine  can  realize  Is  limited  by  the  complexity  of  the  logical 
system,  or  by  the  number  of  A  Units  in  the  Perceptron.  Thus  the 
capability  of  a  Perceptron  for  discovering  and  learning  coiiQ>lex 
Input-output  relationships  is  roughly  determined  by  the  number 
of  A  Units  In  the  machine,  and  the  performance  of  a  Perceptron 
will  be  Improved  by  increasing  the  niunber  of  A  Units. 
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2.5  There  le,  of  course,  a  maximum  number  of  A  Units  which 
may  be  used  without  duplication,  due  to  the  finite  number  of 
possible  combinations  of  wiring  between  the  S  Units  and  the 

A  Units.  If  there  are  N  Stimulus  Inputs,  then  there  are  Ni 

nlTB-nTl 


ways  In  which  n  of  them  can  be  connected  to  an  A  Unit.  If  each 
of  the  connections  can  either  be  to  the  direct  output  of  the  3 
Unit  or  to  Its  complemented  output,  then  the  number  of  distinct 
connections  Is  Increased  by  a  factor  of  2^.  Summing  these  over¬ 
all  values  of  n  from  0  to  N,  we  obtain  2^  possible  ways  of  wiring 

A  Units  If  complements  of  the  S  Unit  outputs  are  not  used  and  3^ 
ways  If  coiig>lements  are  allowed.  However,  for  each  A  Unit  with 
n  Input  connections,  there  are  n  possible  settings  of  Its  thresh¬ 
old  which  will  give  distinct  logical  operation.  Thus  n  different 
A  Units  could  be  built  with  the  same  input  connections  but 
differing  In  thresholds,  which  will  give  different  logical  out¬ 
puts.  It  must  be  noted  that  a  given  A  Unit  with  n  Inputs  and  a 
threshold  of  k  will  have  a  logical  output  complementary  to  that 
of  another  A  Unit  with  Its  n  Inputs  connected  to  the  complements 
of  the  same  set  of  n  S  Units  and  with  a  threshold  of  n-k.  Half 
the  A  Units  will  then  have  outputs  complementary  to  the  other 
half,  so  there  are  only  n  ^  NI  logically  Independent 

?  ^  ni  (N-n)I 


A  Units  possible  with  n  Inputs  from  the  set  of  N  Stimulus  Units. 
Summing  this  over  n  from  0  to  N  yields  N  for  the  total  number 

of  different  A  Units  which  may  be  built  Into  an  N  input  Perceptron 
The  number  of  wires  involved  in  the  connections  from  the  S  Units 
to  the  A  Units  may  also  be  computed  since  there  are  n  wires  to 
each  A  Unit  with  n  Inputs.  Thus  the  number  of  wires  In  the 


S  -  A  matrix  is  n^  ^n  NI  or 

^  3  ®  nl  (R-n)l 
0 


I  (2N+1)  •  3*. 


2.6  The  size  of  these  numbers  can  best  be  appreciated  by 
considering  a  simple  example.  In  a  10  input  machine  there  would 
be  1024  possible  A  Units  if  no  complements  were  allowed  or 
59*049  A  Units  if  S  Unit  outputs  and  their  complements  were 
allowed,  using  fixed  thresholds  in  both  eases.  If  all  logically 
distinct  thresholds  were  allowed  In  order  to  produce  the  most 
general  machine,  there  could  be  as  many  as  196,830  A  Units  used 
without  duplicating  or  conqplementlng  any  logical  functions.  This 
machine  would  require  1,377,810  wires  in  the  connections  between 
S  Units  and  A  Unlts^  While  these  numbers  are  still  very  small 
compared  to  the  2(2*)  or  approximately  lo308  different  logical 
functions  which  it  is  possible  to  form  from  the  10  inputs,  it  is 
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fairly  obvious  that  one  would  not  want  to  build  the  most  general 
Perceptron  for  even  ten  Inputs  using  present  circuit  techniques. 
Thus  It  Is  necessary  to  build  Perceptrons  using  only  a  small 
percentage  of  their  total  possible  A  Units,  generally  limiting 
the  selection  to  some  class  (such  as  a  fixed  value  of  n)  and 
then  taking  only  a  random  sampling  of  that  class.  While  some 
work  has  been  done  on  the  slope  and  asymptote  of  the  learning 
curves  of  Perceptrons  as  a  function  of  the  number  of  A  Units, 

It  is  still  not  clear  Just  what  limits  are  placed  on  the  appor¬ 
tioning  of  the  input  space  by  a  Perceptron  with  a  given  number 
of  A  Units. 


HALP-PERCEPTRON 

3.1  A  simpler  form  of  learning  machine  which  Is  much 
smaller  than  the  Perceptron  has  also  been  proposed,  and  it  la 
termed  a  half -Perceptron  because  It  Is  really  a  degenerate  form 
of  the  Perceptron.  If  a  Perceptron  is  built  with  exactly  N 
Association  Units  connected  to  each  of  the  R  Units,  and  each  of 
the  A  Units  Is  connected  simply  to  one  of  the  S  Units,  the 
machine  reduces  to  the  form  of  Figure  2.  In  this  device  there 
Is  clearly  only  one  logical  transormatlon  from  the  input  space 
(states  of  the  S  Units)  to  the  output  space  (excitations  of  the 
R  Units),  while  In  the  Perceptron  there  are  two  layers  of  logi¬ 
cal  transformation.  The  first  of  these  may  be  considered  as  a 
transformation  from  Input  space  to  A  Unit  excitation  space,  and 
the  second  as  a  transformation  from  A  Unit  space  to  the  output 
space.  The  additional  logical  "depth”  In  the  Perceptron  allows 
It  to  realize  much  more  complex  logical  functions  than  can  the 
half -Perceptron  with  a  logical  depth  of  only  one.  It  seems 
reasonable  to  expect  that  a  machine  similar  to  the  Perceptron 
with  a  logical  depth  of  three  or  more  (obtained  by  two  or  more 
layers  of  A  Units,  with  each  layer  providing  the  excitation  for 
the  next)  would  be  even  more  powerful  than  the  Perceptron. 

3.2  The  half-Peroeptron  with  a  single  R  Unit  Is  a  suf¬ 
ficiently  simple  machine  that  it  Is  not  too  difficult  to  see  the 
restrictions  on  the  functions  that  may  be  realized  by  It. 
Basically  the  device  produces  an  arbitrary  linear  combination  of 
the  input  excitations  and  then  determines  vdiether  this  Is 
greater  or  lees  than  a  fixed  threshold.  In  terms  of  the  parti¬ 
tioning  of  the  input  space,  the  result  Is  that  the  space  will 
consist  of  only  two  regions  or  decision  classes;  and  these  will 
be  separated  by  an  arbitrarily  placed  and  oriented  N-dlmensional 
hyperplane.  This  restricts  the  types  of  functions  which  may  be 
realized  by  the  half -Perceptron,  and  the  restriction  grows  more 
severe  as  the  number  of  Inputs  Increases.  For  example  both  of 
the  two  functions  of  no  Inputs,  all  of  the  four  functions  of  one 
variable,  and  14  of  the  16  functions  of  two  variables  may  be 
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realized;  while  only  IOI4.  of  the  256  functions  of  three  variables 
may  be  realized.  In  the  two-input  case  such  functions  as  the 
logical  ”or”  or  ’’and”  may  be  realized,  while  the  disjunctive 
functions  testing  for  agreement  or  disagreement  In  sign  cannot 
be  realized. 

3.3  In  half-Peroeptrons  with  more  than  one  R  Unit,  the 
same  restrictions  apply  to  each  of  the  R  Units  Individually. 
However,  If  the  several  R  Unit  outputs  are  taken  together  in  some 
logical  fashion  to  represent  a  single  decision  (corresponding  to 
adding  a  second  layer  of  logic),  more  complicated  regions  can  be 
formed  although  they  will  still  be  bounded  by  hyperplanes.  Thus, 
for  example,  the  test  for  agreement  of  two  or  more  Inputs  may  be 
made  as  the  logical  "or**  of  two  R  Unit  outputs;  one  of  which  pro¬ 
duces  the  ”and"  of  the  required  Inputs  and  the  other  of  which 
produces  the  ’’not-or'*  of  the  Inputs. 


COMPUTER  SIMULATION  OP  THE  PERCEPTRON 

I;..!  The  IBM  70I4.  computer  simulation  program  for  the  Per- 
ceptron  Is  given  in  Appendix  1.  The  program  was  written  in  SHARE 
symbolic  language  rather  than  in  FORTRAN  In  order  to  provide  more 
flexibility  In  coding  for  decreased  running  time.  Running  of  the 
program  Is  Intended  to  be  under  control  of  the  BELL  Operating 
System,  and  the  modified  FORTRAN  Input  and  output  routines 
available  as  part  of  the  BELL  system  were  used  for  all  input  and 
output  routines  except  for  binary  tape  operations.  Because  of 
BELL  system  use  of  sense  switches,  program  stops  were  necessary 
within  the  main  program  In  order  to  allow  resetting  of  the  sense 
switches  after  leaving  and  before  re-entering  BELL  system  control. 

k.,2  The  number  of  S  Units  for  the  Ferceptron  was  chosen  to 
be  6I4.,  and  the  number  of  R  Units  can  be  varied  from  one  to  six 
as  determined  by  the  Initial  parts  of  the  Ferceptron  program. 

The  number  of  A  Units  was  then  selected  to  be  as  large  as  possible, 
consistent  with  machine  size  and  operating  speed.  The  most  criti¬ 
cal  part  of  the  program  for  speed  considerations  Is  the  computa¬ 
tion  of  the  A  Unit  excitations  from  the  S  Unit  states  since  this 
requires  a  determination  of  all  the  S  to  A  connections  for  each 
Input  data  sample.  Because  of  the  large  number  of  connections 
this  mu8t.be  done  In  the  fastest  way  possible,  even  at  the  sacri¬ 
fice  of  storage  space.  Repeated  computation  of  these  connections 
Is  out  of  the  question,  and  storage  on  tape  Is  also  too  slow  to 
be  practical.  Even  though  the  address  of  the  S  Unit  has  only  six 
significant  bits.  It  Is  unwieldy  to  store  several  addresses  In 
the  same  word  of  core  storage  because  of  the  computation  neces¬ 
sary  to  select  any  one  of  them.  Thus  the  only  alternative  was 
to  reserve  one  word  of  core  storage  for  each  connection  between 
S  and  A  Units.  Approximately  22,000  words  were  available  for 
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this  fuziotlon«  and  these  were  divided  among  2760  A  Units  where 
each  A  Unit  would  have  6  Input  oonneotlons.  Thus  the  parameters 
of  the  Peroeptron  are  64  Stimulus  Units,  1  to  6  Response  Units, 
and  2760  Association  Units  equally  divided  among  the  Response 
Units,  with  each  A  Unit  having  8  connections  to  S  Units. 

4*3  The  Flow  Chart  representing  the  Peroeptron  operation 
Is  shown  In  Figure  3.  The  first  operation  is  an  HPR  to  allow 
the  sense  switches  to  be  set  as  desired  for  starting  the  program. 
This  Is  followed  by  a  test  of  sense  switch  4*  this  switch 
Is  down,  binary  tape  7  Is  read  to  Insert  a  new  set  of  S  to  A 
cbnnections  (essentially  rewire)  In  the  Perceptron.  If  the 
switch  Is  up,  binary  tape  8  Is  read  Instead,  which  contains  the 
last  previously  used  set  of  S  to  A  connections  as  well  as  the 
last  previous  A  Unit  Values  and  other  Perceptron  parameters. 

If  tape  8  was  read,  sense  switch  3  Is  tested  next.  If  the 
switch  Is  down  a  card  (or  a  record  on  tape  0,  depending  on  the 
setting  of  sense  switch  5)  is  read  containing  6  parameters 
defining  the  Peroeptron  operation.  PLUS  and  MIHUS  control  the 
steps  taken  In  A  Unit  Values  during  the  training  process  and  may 
be  used  to  modify  the  training  scheme.  THOLD  Is  the  fixed 
threshold  level  for  all  the  A  Units.  INOEX4  contains  R,  the 
desired  number  of  Response  Units,  and  IKDEX5  contains  the  number 
2760/R,  or  the  number  of  A  Units  connected  to  each  R  Unit. 

ZROVAL  Is  a  control  word  which  Is  tested  in  the  next  step  of  the 
program.  If  ZROVAL  Is  zero,  all  the  A  Unit  Values  In  the 
machine  are  Initialized  at  zero  Value  or  at  the  Values  read  in 
from  tape  6.  If  ZROVAL  Is  not  zero.  Initial  Values  are  then 
read  In  from  cards  or  tape  0.  If  sense  switch  3  Is  up,  the 
machine  parameters  and  A  Unit  Values  are  left  as  they  were  zNiad 
in  from  tape  8.  The  machine  parameters  are  then  printed  ont 
and  the  program  stops  to  allow  sense  switch  5  to  be  reset  or  to 
allow  tape  0  to  be  changed.  The  Perceptron  is  now  Initialized 
and  ready  to  begin  reading  data. 

4<4  64  S  Unit  excitations  and  the  correct  R  Unit 

responses  corresponding  to  one  data  point  are  then  z*ead  In  from 
a  card  or  a  record  on  tape  9.  The  first  64  card  columns  contain 
the  S  Unit  Inputs,  then  after  two  blank  spaces  columns  67-72 
contain  the  R  Unit  correct  answers.  If  only  R  of  the  six  R  Units 
are  used,  then  the  R  correct  answers  should  be  put  In  columns 
73‘‘R  to  72.  Although  the  machine  was  primarily  designed  for 
binary  Inputs  so  the  S  Unit  excitations  would  be  I's  or  0*s,  It 
Is  possible  to  use  any  number  from  0  through  9  for  the  excita¬ 
tion.  Using  multi-level  data  of  this  sort  tends  to  emphasize 
the  data  points  with  the  highest  levels  at  any  sample  and  Ignore 
the  lower  level  li^uts.  This  may  be  desirable  for  some  types  of 
problems  but  not  for  others. 
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The  next  section  of  the  program  Is  the  most  critical. 

In  that  the  computer  spends  practically  all  Its  time  executing 
one  small  loop.  The  five-instruction  loop  Is: 

LDQ  CONMAT-t-aaOdO,  1 

STQ 

ADD 

TXI  *+1,1,1 
TIX 

Before  each  entry  Into  the  loop  Index  Register  Is  loaded  with 
an  6  and  the  accumulator  is  cleared.  The  22000  words  of  the 
CONMAT  matrix  contain  +0i4.00000004xx  (or  possibly  +0l4.02000004xx) 
where  xx  represents  a  random  octal  number  corresponding  to  one 
of  the  S  Units.  The  first  two  Instructions  In  the  loop  move  the 
word  In  CONMAT  (under  control  of  Index  Register  1)  Into  the  next 
executable  location  taking  It  by  way  of  the  MQ.  There  the  word 
Is  Interpreted  as  an  ADD  (or  SUB)  Instruction  so  that  the  con¬ 
tents  of  the  XX  S  Unit  are  added  to  (or  subtracted  from)  the 
accumulator.  The  next  Instruction  Increments  IR  #1,  and  the 
last  testa  IR  #[4.  and  transfers  out  of  the  loop  after  8  Itera¬ 
tions.  The  contents  of  the  accumulator  then  equal  the  excitation 
to  an  A  Unit.  The  threshold  is  subtracted  and  the  net  excitation 
Is  stored  In  an  A  Unit  memory  location.  The  A  Units  are  counted 
by  IR  #2;  and  the  program  loops  back,  re-lnltiallzes  IR  #4  and 
the  accumulator,  and  computes  another  A  Unit  excitation  until  all 
2760  A  Units  have  been  handled.  Thus  the  5-lnstructlon  loop  is 
Iterated  a  total  of  22000  times,  and  the  remaining  five  instruc¬ 
tions  In  the  larger  loop  2760  times,  for  each  data  Input  sample. 
The  number  of  machine  cycles  for  this  part  of  the  program  Is  then 
2Q!(22060)+5(2760)j  or  248,400  cycles.  This  requires  almost  3 
seconds  of  machine  time,  or  considerably  more  tlMn  half  the  total 
5.1  second  computing  time  required  for  each  data  input. 

4*6  Next  the  A  Unit  excitations  are  tested  for  sign,  and 
their  state  (On  If  sign  was  plus.  Off  If  minus)  Is  stored.  The 
Values  of  the  On  A  Units  are  added,  and  the  Values  of  the  Off  A 
Units  are  subtracted  to  compute  the  excitations  to  the  R  Units. 
Then  the  R  Unit  excitations  are  tested  for  sign,  and  the  R  Unit 
is  put  in  Its  "1"  state  if  the  sign  Is  plus  and  "0"  If  the  exclta 
tlon  Is  negative.  This  completes  computation  of  the  Ferceptron 
output  decision;  It  Is  now  necessary  to  check  the  decisions 
against  the  correct  answers  and  to  train  the  machine. 

4>7  Each  R  Unit  state  Is  compared  with  the  answer  supplied 
on  the  data  card,  and  a  zero  Is  placed  In  a  checking  storage  to 
Indicate  a  correct  answer.  If  the  answer  Is  wrong,  a  "1"  Is 
stored  and  sense  switch  1  Is  tested  to  determine  whether  the 
machine  Is  to  be  corrected  or  not.  If  switch  1  Is  up,  no  adjust¬ 
ment  is  made  to  the  Ferceptron.  If  the  switch  Is  down,  the 
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A  Unit  values  are  to  be  corrected  In  such  a  way  as  to  Increase 
the  probability  of  a  correct  answer.  The  adjusting  (or  train¬ 
ing)  logic  Is  as  follows:  If  the  output  of  an  R  Unit  was  zero 
when  the  desired  output  was  a  ”1",  then  It  is  necessary  to 
Increase  the  excitation  to  the  R  Unit.  This  Is  done  by  adding 
a  fixed  step  of  magnitude  PLUS  to  the  Values  of  all  the  A  Units 
which  are  ^'on"  and  which  are  connected  to  the  given  R  Unit.  A 
fixed  step  equal  to  MINUS  Is  also  added  to  the  Values  of  the  A 
Units  which  are  "off^'  and  are  connected  to  the  given  R  Unit. 

Thus  the  Values  of  the  "on”  A  Units  are  Increased,  and  the  Values 
of  the  "off"  A  Units  are  decreased,  both  of  which  have  the 
effect  of  increasing  the  excitation  to  the  R  Unit.  If  the  R 
Unit  output  Is  ”1”  and  the  desired  output  is  ”0”,  then  it  Is 
necessary  to  decrease  the  excitation  to  the  R  Unit.  This  Is 
done  by  subtracting  PLUS  from  the' Values  of  the  "on”  A  Units 
and  subtracting  MINUS  from  the  "off”  A  Units.  No  changes  are 
made  In  A  Unit  Values  if  the  R  Unit  output  agrees  with  the 
desired  response.  By  adjusting  the  relative  sizes  of  PLUS  and 
MINUS,  It  is  possible  to  achieve  different  sorts  of  reinforcement 
behavior.  Notice  that  the  training  here  Is  only  in  the  direction 
of  correcting  the  error  and  takes  place  in  fixed  steps.  This 
differs  from  the  procedure  usually  used  at  Cornell  where  the 
correction  is  made  big  enough  to  force  the  correct  response  be¬ 
fore  the  machine  goes  on  to  the  next  data  sample..  In  either 
case  a  number  of  runs  through  a  given  list  of  data  points  are 
usually  necessary  before  perfect  leai*hing  occurs,  and  the  choice 
as  to  which  of  the  techniques  Is  preferable  depends  primarily  on 
the  application  for  which  the  Perceptron  Is  Intended. 

4.8  After  the  Perceptron  has  responded  to  an  Input  data 
sample  and  has  been  trained  as  necessary,  a  single  line  of  out¬ 
put  is  written  summarizing  these  operations.  The  first  64  entries 
in  the  line  identify  the  Input  data  point,  and  the  following  six 
bits  correspond  to  the  desired  responses  as  read  from  the  Input 
card  (or  tape).  The  next  group  of  six  bits  represents  the  R 

Unit  states,  and  the  last  group  of  six  bits  indicates  agreement 
(a  zero)  or  disagreement  (a  one)  of  the  R  Unit  states  and  the 
desired  outputs.  A  single  bit  at  the  end  of  the  line  Indicates 
the  state  of  sense  switch  1;  being  a  ”1”  if  the  Perceptron  was 
being  trained  or  ”0"  If  the  machine  was  being  tested  without 
training.  This  output  can  be  written  only  on  tape  9  If  sense 
switch  6  Is  up,  or  on  both  tape  9  and  the  on-line  printer  If 
the  switch  Is  down. 

4.9  Finally  sense  switch  2  Is  tested  as  a  method  of  ending 
the  computer  run.  If  switch  2  Is  up,  the  computer  reads  another 
input  card  (or  tape  record)  and  repeats  Its  operations  on  the 
next  data  sample.  \Vhen  switch  2  Is  put  In  its  down  position, 
the  computer  will  stop  reading  Input  data  and  transfer  to  an  un¬ 
loading  routine.  This  routine  prints  out  the  Values  of  all  the 
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A  Units  on  the  output  tape  (tape  9)  and/or  the  printer;  then  it 
stores  the  S  to  A  connection  matrix  (CONMAT),  the  A  Unit  Values, 
and  the  other  Peroeptron  parameters  on  binary  tape  8  to  allow 
temporary  suspension  of  running  while  maintaining  stored  Values 
and  other  parameters.  Notice  that  the  A  Units  are  stored  back¬ 
wards  in  the  machine.  Thus  the  A  Unit  in  the  lowest  machine 
address  (and  whose  Value  is  first  in  the  printout)  is  connected 
to  the  last  R  Unit  (whose  outputs  appear  in  the  sixth  bit  of  the 
R  Unit  grot;qp  in  the  printout),  and  its  inputs  are  represented  by 
the  last  eight  entries  in  the  CONMAT  list.  Before  returning  to 
the  BBLL  System,  the  machine  stops  once  more  to  allow  the  sense 
switches  to  be  returned  to  their  normal  positions.  Then  an 
automatic  return  to  BELL  System  control  allows  sequencing  of  the 
next  Job  into  the  computer. 


QENERATION  OF  RANDOM  PEROEPTRON  CONNECTIONS 

5>1  Since  the  Perceptron  program  requires  as  one  of  its 
inputs  a  list  of  the  S  to  A  cross-connections  (CONMAT),  a  pro¬ 
gram  was  written  to  generate  this  connection  matrix.  The 
FORTRAN  3  program  shown  In  Appendix  2  generates  this  list  from 
a  random  number  subroutine  and  writes  the  CONMAT  list  on  binary 
tape  7*  The  connections  to  each  A  Unit  are  independent  of  those 
to  other  A  Units,  but  the  eight  Inputs  to  each  A  Unit  are 
selected  such  that  no  two  of  them  coxmect  to  the  same  S  Unit. 

The  particular  random  set  of  connections  chosen  depends  on  the 
octal  number  Q  which  is  fed  in  to  start  the  routine. 

5*2  When  the  octal  number  INST  on  the  input  card  repre¬ 
sents  ADD  400,  the  word  stored  on  the  binary  tape  corresponding 
to  each  entry  in  the  CONMAT  list  is  -fO^OOOOOOOl+xx,  where  the  xx 
represents  the  octal  number  (0  to  77)  identifying  the  S  Unit 
Involved  in  the  connection.  The  Perceptron  program  translates 
this  by  adding  the  output  of  S  Unit  xx  (octal)  to  the  excitation 
of  the  proper  A  Unit.  This  simulates  the  operation  of  a 
Perceptron  using  only  the  uncomplemented  outputs  of  the  S  Units. 
It  is  possible  to  modify  the  CONMAT  program  slightly  to  simulate 
both  the  S  Unit  outputs  and  their  complements  by  mixing  both 
ADD  and  SUB  instructions  in  the  CONMAT  tape.  Thus  the  words  on 
the  binary  tape  would  be  either  of  the  form  -t'Oli.OOOOOOOUxx  or 
alternately  ‘^0402000004xx,  the  form  used  for  each  connection  to 
be  selected  systematically  or  at  random. 

5*3  A  printout  list  of  the  connections  is  also  provided 
for  reference,  where  a  list  of  eight  decimal  numbers  (0  to  63) 
is  given  for  each  A  Unit,  identifying  the  S  Units  to  which  it 
is  connected.  This  list  is  in  the  same  order  as  the  binary 
list,  and  so  the  first  group  of  eight  numbers  in  the  CONMAT 
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Hating  corresponds  to  the  last  A  Unit  In  the  Feroeptron  pro¬ 
gram.  The  first  eight  octal  Instruction  words  are  also  printed 
on  line  as  a  checking  feature  on  the  program. 


COMPUTER  SIMULATION  OP  THE  HALP-PERCEPTRON 

6.1  The  half -Feroeptron  program  shown  In  Appendix  3 
Is  designed  to  have  Inputs  and  outputs  compatible  with  the 
Feroeptron  program.  It  too  has  624.  S  Units  and  6  R  Units,  but 
its  logical  behavior  Is  governed  by  the  coefficients  stoirad  in 
Its  624jc6  element  3  to  R  matrix.  Because  of  its  simpler  logical 
structure,  the  half -Feroeptron  rune  at  the  rate  of  approximately 
0.5  second  per  data  sample  as  opposed  to  the  ^.1  seconds  per 
sample  for  the  Feroeptron.  It  also  becomes  practical  to  print 
out  the  Internal  structure  (MATRIX)  of  the  half -Feroeptron 
periodically  during  training. 

6.2  Figure  24.  shows  the  flow  chart  of  the  half -Feroeptron 
program.  As  In  the  Ferceptron,  the  first  Instruction  that  is 
encountered  after  control  Is  transferred  from  the  BELL  system 
Is  a  STOF  to  allow  the  sense  switches  to  be  set  for  the  program. 

If  sense  switch  24.  la  up,  the  MATRIX  is  read  In  from  binary  tape  8, 
which  may  contain  the  MATRIX  elements  from  a  preceding  run.  If 
the  switch  Is  down,  sense  switch  3  is  then  tested  to  Initialise 
the  MATRIX.  If  switch  3  la  up,  all  MATRIX  elements  are  assumed 

to  he  zero;  If  the  switch  Is  down,  the  MATRIX  Is  read  Into  the 
machine  from  the  card  reader.  The  half-Ferceptron  Is  now 
Initialized  and  ready  to  read  Input  data. 

6.3  The  input  data  sample  and  the  desired  responses  are 
then  read  from  a  card  (If  switch  5  la  down)  or  from  tape  0  (if 
switch  5  la  up)  In  the  same  format  as  that  used  In  the  Feroeptron. 
The  only  significant  difference  Is  that  the  half-Ferceptron 
always  assumes  that  six  responses  are  required,  and  blank  spaces 
on  the  ANSWER  field  of  the  card  will  be  Interpreted  as  required 
zeros.  The  only  penalty  paid  for  tne  assumed  six  R  Units  In  the 
half -Feroeptron  Is  a  small  time  loss.  In  the  Feroeptron  It  was 
definitely  advantageous  to  divide  iqp  the  available  A  Units  among 
only  the  required  number  of  R  Units. 

6.14.  The  R  Unit  states  are  determined  by  multiplying  each 
S  Unit  excitation  by  the  appropriate  MATRIX  element  and  detemln- 
Ing  whether  the  net  excitation  to  the  R  Unit  Is  positive  or  nega¬ 
tive.  The  cheeking  and  training  sections  are  essentially  the 
same  as  those  In  the  Ferceptron.  Each  R  Unit  state  Is  compared 
with  the  desired  response,  and  the  results  of  the  comparison 
stored  in  the  CHECK  locations.  If  an  R  Unit  state  Is  wrong, 
sense  switch  1  Is  tested  to  determine  whether  the  MATRIX  elements 
should  be  adjusted.  If  the  switch  Is  down,  the  training  pro- 
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oedure  Is  as  follows.  If  the  R  Unit  state  Is  zero  and  the 
desired  response  is  a  "I",  then  the  excitation  to  the  R  Unit 
should  be  Increased.  Thus  the  MATRIX  elements  associated  with 
that  R  Unit  are  each  increased  by  the  amount  of  excitation  to 
the  associated  S  Unit.  The  reason  for  this  is  that  S  Units  which 
have  zero  excitation  are  not  affecting  the  R  Unit  excitation, 
and  there  is  no  justification  for  increasing  the  strength  of 
their  connections  to  the  R  Unit.  If  the  S  Unit  excitations  are 
allowed  to  range  from  0  through  9  (rather  than  just  being  binary 
inputs),  then  the  S  Units  having  the  largest  excitation  will  have 
their  oohnections  strengthened  the  most.  There  is  some  justifi¬ 
cation  for  this,  although  the  training  procedure  is  designed 
primarily  for  binary  inputs.  If  the  R  Unit  state  is  a  "1”  where 
a  ’’O”  is  desired,  the  excitation  should  be  decreased.  Thus  each 
MATRIX  element  is  decreased  by  the  amount  of  the  excitation  to 
the  associated  3  Unit.  If  the  R  Unit  state  agrees  with  the 
desired  response,  no  change  is  made  in  the  MATRIX  elements. 

6.5  The  printout  after  each  data  sample  la  processed  is 
Identical  to  that  of  the  Ferceptron,  containing  the  6I4.  S  Unit 
excitations,  the  6  desired  responses,  the  6  R  Unit  states,  the 
6  checking  results,  and  the  single  bit  indicating  whether  the 
half -Ferceptron  was  being  trained  or  merely  tested.  Output  is 
on  tape  9*  and  may  also  be  printed  on  line  by  placing  sense 
switch  6  down.  After  each  data  sample  is  processed  and  the  re¬ 
sults  printed,  sense  switch  2  is  tested  to  determine  whether  the 
next  data  sample  should  be  read  (up)  or  the  running  terminated 
(do%fn).  If  the  switch  is  down  the  MATRIX  elements  are  printed 
out  on  tape  9  (and  on  line  if  switch  6  is  down),  and  are  also 
listed  in  binary  on  tape  8  so  they  can  be  stored  for  later  re¬ 
sumption  of  running.  The  program  then  stops  to  allow  the  sense 
switches  to  be  reset,  and  then  transfers  back  to  the  BELL  System. 
During  the  running  of  the  half-Psrosptron,  the  MATRIX  is  also 
printed  out  on  tape  9  (and  optionally  on  line)  after  each  100 
data  samples  have  been  processed.  After  this  printout  the 
machine  continues  running.  The  MATRIX  list  is  always  used  and 
printed  in  the  same  order,  so  the  first  6I4.  entries  in  the  list 
correspond  to  the . connections  from  the  S  Units  (in  asosnding 
numerical  order)  to  the  first  R  Unit. 


EXPERIMENTAL  EVALUATION 
OF  THE  FERCEPTRON  AND  HALF -FERCEPTRON 

7*1  An  experimental  problem  was  desired  to  test  the  capa¬ 
bilities  of  the  Ferceptron  and  the  ha  If -Ferceptron.  The  sin^lest 
logical  problem  that  can  be  given  a  classification  machine  is 
to  require  that  each  of  its  outputs  be  Identical  to  one  of  its 
inputs,  and  thus  independent  of  all  other  inputs.  This  was  the 
test  problem  used  for  the  two  learning  machines,  and  the  states 
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of  the  six  R  Units  were  required  to  be  Identical  to  the  states 
of  the  first  six  S  Units.  The  FORTRAN  program  in  Appendix  I4. 
generates  the  input  data  for  the  problem.  A  random  list  of  64 
binary  numbers  is  generated  from  the  random  number  routine  and 
used  as  the  S  Unit  excitation.  The  first  six  of  these  are  also 
used  as  the  desired  responses,  and  the  two  groups  are  written 
on  tape  6  in  the  proper  format  for  input  to  the  Ferceptron  or 
half-Ferceptron  programs.  A  list  of  1000  data  saBqsles  was  pre¬ 
pared  in  this  way. 

7.2  A  measure  of  the  degree  of  learning  is  given  by  the 
number  of  errors  occurring  in  a  fixed  number  of  data  samples. 

The  errors  were  counted  in  groups  of  ten  input  samples,  and  the 
number  of  errors  in  each  group  is  plotted  as  a  function  of  the 
total  number  of  input  samples  in  Figure  5  for  the  half-Ferceptron 
and  Figure  6  for  the  Ferceptron.  In  the  half-Ferceptron  case 
all  1000  input  samples  were  processed  in  direct  succession.  In 
the  Perception  the  first  237  samples  were  processed  in  one  run, 
then  in  the  second  run  the  first  520  samples  were  processed. 

Thus  between  input  samples  238  and  474  machine  was  process¬ 
ing  input  data  that  it  had  already  seen  once  before,  while  other 
data  saoqples  were  only  being  processed  for  the  first  time.  The 
fact  that  there  is  no  noticeable  discontinuity  in  the  learning 
curve  either  at  the  237  or  the  474  sample  points  verifies  the 
fact  that  the  learning  process  chosen  is  only 'an  incremental 
training  and  that  more  than  one  pass  through  a  given  set  of  data 
is  necessary  before  it  is  fully  assimilated. 

7.3  The  learning  curves  for  the  half-Ferceptron  and  the 
Ferceptron  are  similar  in  general  form,  beginning  from  a  point 
in  the  vicinity  of  the  30  errors  (in  sixty  possible  bits)  that 
would  be  predicted  from  purely  random  operation  and  decaying 
more  or  less  exponentially  toward  some  lower  error  rate.  The 
initial  slopes  for  the  two  machines  are  roughly  the  same,  al¬ 
though  the  half -Ferceptron  reaches  a  higher  degree  of  learning 
much  sooner  than  the  Ferceptron.  Notice  that  in  the  half-Ferceptron 
the  error  rate  gets  quite  small,  and  that  the  machine  actually 

made  no  errors  at  all  in  the  last  100  samples.  The  Ferceptron, 
on  the  other  hand,  only  approaches  an  error  rate  of  about  4 
errors  per  60  bits  and  seems  to  maintain  this  level  of  learning. 

7.4  The  difference  in  final  learning  ability  of  the  two 
machines  is  due  to  the  difference  in  logical  organisation  of  the 
machines,  and  to  the  special  nature  of  this  particular  problem. 

What  is  really  desired  in  this  problem  is  that  the  oonnections 
in  the  logical  network  from  S  Unit  X  to  R  Unit  X  be  strengthened, 
and  that  the  connections  from  all  the  other  S  Units  to  that  R 
Unit  be  zero  or  canceled.  Because  of  the  organization  of  the 
half-Ferceptron,  this  can  actually  be  done  in  this  machine;  and 
in  fact  it  can  be  seen  directly  from  the  MATRIX  elements.  In 
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the  Perceptron,  however,  each  path  from  a  given  S  Unit  to  a 
particular  R  Unit  must  pass  throu^  an  A  Unit  to  which  other  S 
Units  are  connected.  Thus,  while  the  values  of  all  A  Units  which 
do  not  provide  transmission  paths  between  the  desired  S  and  R 
Units  can  be  made  zero,  it  is  impossible  to  eliminate  the  noise 
produced  by  the  other  S  Unit  excitations  as  they  pass  through  A 
Units  which  do  provide  transmission  paths  for  the  desired  S  Unit. 
This  noise  can  only  be  reduced  by  the  statistical  averaging  from 
a  large  number  of  A  Units  and  goes  down  as  the  square  root  of 
the  number  of  A  Units.  Thus  the  half-Perceptron  can  have  a  zero 
error  rate  for  this  simple  class  of  problem  but  the  Perceptron 
organization  limits  the  machine  to  some  non-zero  error  rate 
determined  by  the  number  of  A  Units  in  the  machine. 

7.5  Another  significant  difference  between  the  two  machines 
is  the  difficulty  in  Intexpreting  the  information  stored  in  the 
Values  of  the  Perceptron  or  in  the  MATRIX  of  the  half -Perceptron. 
In  the  sample  problem  where  six  of  the  334  connections  in  the 
half-Perceptron  were  to  be  strong  relative  to  the  others,  this 
result  shows  up  very  clearly  in  the  MATRIX  printout.  After  the 
1000  training  samples  the  six  desired  connections  had  coefficients 
ranging  between  31  And  33,  while  the  remaining  378  elements  of 
the  MATRIX  had  an  average  value  of  -0.541  And  a  standard  devia¬ 
tion  around  this  average  of  1.43*  ^he  negative  average  value  of 
the  weak  connections  is  due  to  the  tendency  for  the  average  of 
the  MATRIX  elements  to  be  zero.  In  the  Perceptron  the  connec¬ 
tions  between  the  desired  S  and  R  Units  (and  thus  the  learning) 
are  distributed  throu£^out  the  machine.  In  this  problem  the 
learning  can  be  demonstrated  by  dividing  the  A  Units  belonging 
to  a  given  R  Unit  into  two  classes;  those  which  connect  to  the 
desired  3  Unit,  and  those  which  do  not.  This  was  done  for  the 
A  Units  connected  to  R  Unit  #5  and  the  distributions  of  the 
Values  plotted  in  Figure  7  for  each  of  the  two  classes.  Neither 
distribution  differs  significantly  from  the  normal  distribution. 
The  mean  of  the  Values  of  those  A  Units  (393)  which  do  not  con¬ 
nect  to  S  Unit  #5  is  -1.6  and  the  standard  deviation  of  the  Values 
is  4*7*  The  mean  for  the  65  units  which  did  connect  to  S  Unit  #5 
was  14*3  And  the  standard  deviation  was  5*0.  The  two  standard 
deviations  are  not  significantly  different,  but  the  different 
means  show  that  a  significant  difference  has  appeared  between 
the  two  classes  of  A  Units,  indicating  that  learning  has  taken 
plAce.  The  standard d»viatlons  are  also  significantly  smaller 
than  v?6  or  6.72,  which  would  be  predicted  by  random  walk  con¬ 
siderations,  indicating  that  the  Values  are  converging  toward 
some  fixed  distribution  and  will  not  continue  to  vary  indefinite¬ 
ly.  In  more  conqplicated  problems  the  learning  will  be  even  mors 
deeply  embedded  in  the  A  Unit  Values,  and  in  practical  cases  it 
may  be  ia^osslble  to  recognize  the  various  classes  of  A  Units 
from  the  distribution  of  their  Values. 
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7.6  The  sample  problem  just  discussed  is  a  very  simple  one, 
and  Is  rather  easily  solved  by  both  the  Perceptron  and  the  half- 
Peroeptron.  A  second  problem  was  designed  In  order  to  demonstrate 
the  limits  of  capability  of  the  two  machines.  In  this  problem 
only  two  R  Units  were  used,  and  In  the  Peroeptron  the  A  Units 
were  divided  between  these,  giving  13d0  A  Units  to  each  R  Unit. 

The  desired  outputs  of  the  machines  were  defined  as  follows: 

If  more  of  the  first  five  S  Units  were  in  the  one  state  than  the 
number  of  the  next  five  S  Units  which  were  In  the  one  state,  the 
output  of  R  Unit  #5  should  be  a  one.  Otherwise  (more  ones  In 
the  second  group  of  five  than  In  the  first  group  of  five,  or  an 
equal  number  In  each)  R  Unit  #5  should  be  zero.  R  Unit  #6  was 
required  to  be  **1**  If  the  number  of  ones  In  the  first  six  S  Units 
was  odd  and  zero  If  the  number  of  ones  was  even.  The  require¬ 
ment  on  R  Unit  #5  Is  still  among  the  class  of  problems  which  may 
be  solved  by  a  half-Perceptron,  and  Is  also  within  the  capabili¬ 
ties  of  the  Peroeptron,  where  again  the  Perceptron  is  likely  to 
exhibit  more  random  noise  In  Its  output  because  of  Its  organiza¬ 
tion.  The  required  solution  for  R  Unit  #6,  however.  Is  one  of 
the  moat  difficult  switching  functions  of  six  variables;  and  It 
may  be  shown  that  It  Is  beyond  the  theoretical  capability  of  the 
> half-Perceptron.  While  a  sufficiently  large  and  general  Peroeptron 
can  handle  this  problem.  It  becomes  very  difficult  for  a 
Perceptron  of  the  size  and  limited  organization  used  here. 

7.7  A  tape  was  made  from  the  input  tape  for  the  previous 
problem,  substituting  new  required  R  Unit  responses  according  to 
the  above  definitions.  The  PORTRAH  3  program  used  to  produce 
this  tape  Is  shown  in  Appendix  $.  This  tape  was  processed  by 
both  the  half -Peroeptron  and  the  Perceptron,  and  the  learning 
curves  are  shown  In  Figures  8  and  9  respectively.  The  errors  on 
each  of  the  two  R  thilts  are  plotted  separately,  and  the  vertical 
axis  on  the  plots  is  the  number  of  errors  on  the  given  R  Unit  In 
each  group  of  20  samples.  The  upper  curve  in  each  Figure  repre¬ 
sents  the  errors  In  R  Unit  #5  and  the  lower  Is  for  R  Unit  #6. 

The  error  rate  for  R  Unit  #5  In  the  half-Perceptron  is  around  2 
or  3  errors  per  20  samples.  This  of  course  Indicates  that  the 
problem  Is  being  solved  by  the  machine,  although  convergence  Is 
not  as  rapid  as  In  the  earlier  problem  of  Figure  The  errors 
per  20  samples  In  R  tTnlt  #6  continue  to  fluctuate  around  10, 
which  Indicates  that  the  machine  Is  merely  guessing  at  random 
and  that  no  solution  Is  being  formed. 

7.8  Only  the  first  I4.OO  samples  of  the  tape  were  processed 
on  the  Peroeptron,  and  the  errors  are  plotted  In  Figure  9.  Again 
the  errors  In  R  Unit  #5  Are  decreasing,  and  the  error  rate  after 
I4.OO  samples  Is  about  $  per  20  samples  and  not  significantly 
different  from  the  error  rate  In  the  half-Perceptron  after  400 
samples.  Also  the  plot  of  H  Unit  #6  errors  for  the  Peroeptron 
shows  no  Indication  of  learning  after  400  sanqples.  It  Is  not 
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clear  whether  this  rate  would  decrease  with  a  much  larger  number 
of  input  samples,  but  for  this  size  Perceptron  it  is  unlikely 
that  the  final  error  rate  after  a  very  large  number  of  samples 
would  get  very  much  below  10.  It  should  be  observed  that  the 
relative  number  of  required  zeros  to  ones  for  R  Unit  #5  is  ap¬ 
proximately  5  to  3  because  of  the  zero  requirement  when  both 
groiq^s  of  S  Units  have  an  equal  number  of  ones.  Thus  a  learning 
machine  could  have  an  error  rate  as  low  as  7*5  por  20  samples 
merely  by  observing  this  bias  and  guessing  all  zeros.  However^ 
there  is  no  evidence  from  the  data  that  either  machine  tried 
this  tactic. 


CONCLUSIONS 

8.1  The  sample  problems  shown  here  demonstrate  the  type 
of  performance  to  be  expected  from  learning  machines  when  they 
are  given  8liiq>le  problems  as  in  the  first  example,  or  when  they 
are  given  problems  requiring  or  exceeding  their  full  capabili¬ 
ties.  The  fact  that  the  half -Perceptron  reached  a  lower  error 
rate  on  the  first  exaaqple  than  the  Perceptron  did,  and  that  the 
performance  was  similar  to  that  of  the  Perceptron  in  the  second 
exaiiq)le,  should  not  be  considered  an  advantage  of  the  half- 
Perceptron  type  of  construction.  The  reason  is  that  the  half- 
Peroeptron  is  capable  of  solving  only  a  limited  class  of  prob¬ 
lems,  and  that  its  more  direct  organization  allows  a  lower  error 
rate  for  the  problems  for  which  it  is  designed.  The  majority  of 
practical  classification  problems,  particularly  problems  which 
require  recognizing  switching  functions,  are  likely  to  exceed 
the  capabilities  of  the  half -Perceptron;  and  a  learning  machine 
having  the  flexibility  of  the  Perceptron  would  be  required. 

8.2  It  is  possible  that  a  machine  concept  could  be  devel¬ 
oped  having  the  flexibility  of  the  Perceptron  organization,  yet 
avoiding  some  of  the  difficulties  Inherent  in  its  organization. 
However  these  difficulties  tend  to  disappear  as  the  number  of  A 
Units  becomes  very  large  due  to  averaging  out  of  the  random 
noise  components  in  the  A  Units.  It  is  presently  Impractical  to 
cozislder  a  much  larger  number  of  A  Units  in  a  computer  simulation 
because  of  speed  and  storage  limitations,  and  present  electro- 
mechanloal  realizations  of  A  Unit  operation  are  too  bulky  and 
expensive  to  allow  very  large  Ferceptrons.  However,  it  is 
reasonable  to  expect  that  some  sort  of  solid  state  device  could 
be  developed  having  the  logical  and  memory  characteristics  of  an 
A  Unit.  If  such  a  device  could  be  built  along  the  general  lines 
of  multi-hole  magnetic  cores  and/or  thin  film  semi -conductor 
circuitry,  Ferceptrons  could  be  considered  having  10^  or  more 

A  Units.  In  such  machines  very  powerful  logical  behavior  and 
learning  capability  would  be  available  for  solving  various  kinds 
of  classification  problems. 
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8.3  The  use  of  machines  such  as  the  Perceptron  for  naval 
problems  such  as  target  classification  would  be  restricted  to 
laboratory  studies.  It  Is  not  practical  to  use  learning  machines 
as  such  on  board  operational  vessels  because  the  amount  of  Infor¬ 
mation  available  (In  the  form  of  signals  from  Identifiable  tar¬ 
gets)  to  each  vessel  would  not  be  sufficient  to  train  such  a 
machine.  Rather  the  proper  use  of  the  learning  machines  would  be 
to  pool  the  Information  available  from  all  sources  (tape  record¬ 
ings  from  ships  or  shore -based  installations)  and  to  feed  all 
this  Information  to  a  single  learning  machine  for  comparative 
analysis.  In  the  research  phase  of  target  classification  studies, 
the  learning  machines  could  be  used  to  determine  which  character¬ 
istics  of  the  Input  data  were  significant  In  classifying  targets. 
In  later  operational  use  of  such  equipment,  the  laboratory 
machine  would  constantly  modify  its  structure  in  such  a  way  as 
to  maintain  an  optimum  classification  scheme,  based  on  the 
currently  available  Input  data.  Measurements  made  on  the  labora¬ 
tory  machine  would  then  provide  information  for  designing,  or 
perhaps  simply  readjusting,  non-adaptlve  operational  classifiers 
for  the  ships  of  the  fleet. 
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ASSOCIATION 


FIG.  I  PERCEPTRON  BLOCK  DIAGRAM 


FI6.  2  HALF-PERCEPTRON  BLOCK  DIAGRAM 


FK.  3  PERCEPTRON  FLOW  CHART 
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20  INPUT  SAMPLES. PROBLEM  2 
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APPENDIX  1  PERCEPTRON  MAIN  PROGRAM 


ORG  24 
HPR 

SWT  4  TEST  FOR  REBUILD 

TRA  RETAPE 

REM  READ  NEW  CONNECTION  MATRIX 
LXA  IND£X2»1 
RTB  7 

CPY  CONMAT+22080»1 
TlX 

TRA  NEWVAL 

REM  READ  OLD  PARAMETERS  AND  CONNECTION  MATRIX 
RETAPE  LXA  INDEX3»1 
RTB  8 

CPY  VALUE+24846»1 

TlX 

NOP 

SWT  3  TEST  FOR  RESET  VALUES 

TRA  FIRST 

REM  READ  NEW  SYSTEM  PARAMETERS 
NEWVAL  TSX  X INPUT *4 
NTR  F4,»0 
NTR  PLUS 
NTR  MINUS 
NTR  THOLD 
NTR  INDEX4 
NTR  .  INDEX5 
NTR  2R0VAL 
XIT  XXXRET 
CLA  ZROVAL 
TZE  *+4 
TSX  XINPUT»4 
■NTR  F2t»0 

MON  VALUEi fVALUE+2759 

FIRST  TSX  OUTPUT f 4  PRINT  PERCEPTRON  CONSTANTS 

NTR  F4»»0 
NTR  PLUS 
NTR  MINUS 
NTR  THQLD 
NTR  INDEX4 
NTR  INDEX5 
NTR  ZROVAL 
XIT  XXXRET 
LXD  INOEX5»l 
SXD  RIGHT-2il 
SXD  RIGHT+lil 
HPR 

REM  READ  NEXT  STIMULUS  AND  COMPUTE  A  INPUTS 
STIM  TSX  XINPUT»4 
NTR  F3i»0 

MON  SUNITf f ANSWER+5 
LXD  ONEfl 
LXD  INDEX2»2 


LOAD  2760 
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NEXTA 

LXD 

INDEX3»4 

LOAD  8 

CLA 

ZERO 

LDQ 

CONMAT+22080»1 

STQ 

*+l 

ADD 

ADDRESS  COMPUTED  BY  PROGRAM 

TXI 

INCREMENT  CONNECTION 

TIX 

COUNT  inner  loop 

SUB 

THOLD 

STO 

AUNIT+2760»2 

TIX 

NEXTA»2»1 

COUNT  A  UNITS 

REM 

COMPUTE  R  UNIT  INPUTS  AND 

OUTPUTS 

LXD 

ONE»l 

LXD 

INDEX4>2 

LOAD  R 

NEXTR 

LXD 

INDEX5*4 

LOAD  2760/R 

CLA 

ZERO 

TESTA 

LDQ 

AUNlT+2760.1 

TQP 

AON 

SUB 

VALUE+2760*! 

LDQ 

MINUS 

STQ 

AUNIT+2760»1 

TXI 

COUNT  A  UNITS 

TIX 

TESTA»4.1 

TRA 

STORER 

AON 

ADD 

VALUE+2760 #1 

LDQ 

PLUS 

STQ 

AUNIT+2760*! 

TXI 

*+l»l»l 

COUNT  A  UNITS 

TIX 

TESTA»4»1 

STORER 

TPL 

RON 

CLA 

ZERO 

TRA 

*+2 

RON 

CLA 

ONE 

STO 

RUNIT+6»2 

TIX 

NEXTR»2.1 

REM 

CHECK  ANSWERS  AND  TRAIN 

LXD 

IN0EX4»1 

LOAD  R 

LXD 

INDEX5f2 

LOAD  2760/R 

TESTR 

CLA 

RUNIT+61I 

SUB 

ANSWER+6tl 

TZE 

RIGHT 

SWT 

1 

TEST  FOR  TRAINING 

TRA 

RIGHT-1 

LXD 

INDEX5f4 

TMI 

INCRE 

CLS 

AUNIT+2760*2 

ADD 

VALUE+2760»2 

STO 

VALUE+2760*2 

TXI 

*+l»2»-l 

TIX 

*-4i4»l 

COUNT  A  UNITS  FOR  GIVEN  R  UNIT 

TRA 

RIGHT-2 

INCRE 

CLA 

AUNIT+2760*2 

ADD 

VALUE+2760 »2 

STO 

VALUE+2760 »2 

TXI 

*+l»2»-l 

2 
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TIX 

*“4»4#1 

COUNT  A  UNITS  FOR 

GIVEN  R  UNIT 

TXI 

*+l»2i 

CLA 

ONE 

RIGHT 

STO 

CHECK+6*! 

TXI 

*+l»2» 

TIX 

TESTR»1#1 

COUNT  R  UNITS 

CLA 

ONE 

SWT 

1 

TEST  FOR  TRAINING 

CLA 

ZERO 

STO 

TRAIN 

TSX 

OUTPUT *4 

NTR 

FltiO 

MON 

SUNIT»#TRAIN 

SWT 

2 

TEST  FOR  STOP 

TRA 

STIM 

TSX 

OUTPUT »4 

PRINT  PRESENT  VALUES 

NTR 

F2f  »0 

MON 

VALUE* #VALUE+2759 

REM 

WRITE  PARAMETERS  AND  CONNECTION  MATRIX  ON  TAPE 

8 

LXA 

INDEX3>1 

WTB 

8 

CPY 

VALUE‘»-24846f  1 

TIX 

*-1*1*1 

HPR 

TSX 

RETURN*4 

HTR 

*-l 

ZERO 

OCT 

000000000000 

ONE 

OCT 

000001000000 

INOEX2 

OCT 

005310053100 

2760  22080 

IN0EX3 

OCT 

000010060416 

8  24846 

FI 

BCD 

5(10X*64I1*3(3X»611) f3XfIl) 

OCT 

777777777777 

F2 

BCD 

1(1216} 

F3 

BCD 

3(64I1*2X*6I1) 

OCT 

777777777777 

F4 

BCD 

1(616) 

ORG 

256 

OCTAL  400 

SUN  IT 

BSS 

64 

ANSWER 

BSS 

6 

RUN  IT 

BSS 

6 

CHECK 

BSS 

6 

TRAIN 

AUNIT 

BSS 

2760 

VALUE 

BSS 

2760 

CONMAT 

BSS 

22080 

INDCX4 

NUMBER  OF  R  UNITS 

(R) 

ZN0EX5 

2760/R 

PLUS 

MINUS 

THOLD 

ZROVAL 

END 

24 

3 


vntn/tifi<ntnu>o*u>ot 
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APPENDIX  2  PERCEPTRON  CONNECTION  MATRIX  PROGRAM 

DIMENSION  LIST(8»2760) 

READ  lOOfOvINST 
N0«0 

DO  40  J«l»2760 
DO  40  I*l»8 
N*I-1 

10  CALL  NUMBER (Q*R) 

IR»R*64» 

IF(N)70*40»20 
20  DO  30  X-1*N 

IF(  IR-LIST(K»Jn30f  10*30 
30  CONTINUE 
40  LIST(I«J)>1R 

PRINT  110* ( (LIST! l«J)fI«l*8)*J>l *2760) 

LXD  NOtd) 

NTB  7 

50  CLA  LIST*(1) 

ARS  18 
ADO  INST 
STO  LIST*(I) 

CPY  LIST*(n 
TXI*60*(n*l 
60  TXL*50*(I)»22079 
WEF  7 

WRITE  OUTPUT  TAPE  ll»120«aiST(I*l)*l«l»a) 

CALL  RETURN 
70  STOP 

100  F0RMATI2012) 

110  F0RMAT<4(4Xf6l3)) 

120  FORMAT (10X«8012) 

END 


1 
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> 

APPENDIX  3 

HALF-PERCEPTRON 

PROGRAM 

. 

ORG 

100 

4 

HPR 

SWT 

4 

TEST  FOR 

NEW  MATRIX 

TRA 

READS 

SWT 

3 

TEST  FOR 

ZERO  MATRIX 

TRA 

STIM-2 

TSX 

X INPUT #4 

NTR 

F3»»ll 

i 

MON 

MATRlX»iMATRIX+3S3 

1 

i 

TRA 

STIM-2 

READS 

LXD 

INDEXltl 

384 

RTB 

8 

CPY 

MATRIX+384#1 

TIX 

CLA 

HNDRED 

STO 

COUNT 

STIM 

TSX 

XINPUT»4 

NTR 

FlrtO 

MON 

SUNIT»*ANSWER+5 

REM 

COMPUTE  RUNIT 

INPUTS 

AND  OUTPUTS 

LXD 

INDEXld 

384 

LXO 

INOEX2t2 

6 

NEXTR 

LXA 

INDEXI(4 

64 

ST2 

RUNIT+6#2 

LDQ 

SUNIT+64i4 

MPY 

MATRIX+384»1 

ADD 

RUNIT+6»2 

STO 

RUNIT+6»2 

« 

TXI 

TIX 

*-5»4»l 

LDQ 

ONE 

TPL 

#+2 

LDQ 

ZERO 

STQ 

RUNIT+6»2 

TIX 

NEXTRtZfl 

REM 

CHECK  ANSWERS 

AND  TRAIN 

LXD 

INOEXlf 1 

364 

LXD 

INDEX2f2 

6 

TESTR 

CLA 

RUNIT+6»2 

SUB 

ANSWER+SfZ' 

TZE 

RIGHT 

SWT 

1 

TEST  FOR 

TRAINING 

TRA 

RIGHT-1 

LXA 

IND£Xlf4 

64 

TMI 

INCRE 

CLA 

MATRlX+384*! 

SUB 

SUNIT+64»4 

• 

STO 

MATRIX+384#1 

TXI 

#+lflf-l 

TIX 

*-4»4»l 

TRA 

RIGHT-2 

INCRE 

CLA 

MATRIX+384»1 

i 

t  ______ _ 
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ADD  SUNIT+64»4 
StO  MATRIX+384»1 
TXI 

Tlx  #-4»4»l 
TXI  »+l»lf64 
CLA  ONE 

RIGHT  STO  CHECK+612 
TXI  *+l»l»-64 
TlX  TESTR»2»1 
CLA  ONE 
SWT  1 
CLA  ZERO 
STO  TRAIN 
TSX  0UTPUTf4 
NTR  F2»f0 
MON  SUNIT»#TRAIN 
CLA  COUNT 
SUB  ONE 
STO  COUNT 
TZE  #+3 
SWT  2 
TRA  STIM 
TSX  OUTPUT »4 
NTR  F4#»0 

MON  MATRlX»iMATRlX+383 
SWT  2 

TRA  STIM-2 
LXD  INDEXlil 
WTB  6 

CPY  MATRIX+384»1 

TlX 

HPR 

TSX  R£TURN»4 
HTR  *-l 

ZERO  OCT  000000000000 
ONE  OCT  000001000000 
HNDRED  OCT  000144000000 
INDEXl  OCT  000600000100 
IN0EX2  OCT  000006000000 
FI  BCD  3(64llt2Xf611) 

OCT  777777777777 
F2  BCD  5(10Xf64Il*3(3Xt6Il) 

OCT  777777777777 
F3  BCD  1(816) 

F4  BCD  1(1616) 

SUNIT  BSS  64 
ANSWER  BSS  6 
RUNIT  BSS  6 
CHECK  BSS  6 
TRAIN 
COUNT 

MATRIX  BSS  384 
END  100 


TEST  FOR  TRAINING 


TEST  FOR  STOP 
PRINT  PRESENT  MATRIX 

TEST  FOR  STOP 

384 


100 

384  64 

6  0 

3XtIl) 


2 
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APPENDIX  4  DAtA  GENERATION  PROBLEM  1 

DIMENSION  KLM(64) 

READ  100*0 
DO  20  J«l*1000 
DO  10  K«l*64 
CALL  NUMBER  (0*R) 

10  KLM(K)«R4>«5 

20  WRITE  OUTPUT  TAPE  6*110*  (KLMIIC)  •K«l  *64)  •( KtM(K)  »K«1*6) 
END  FILE  6 
100  FORMAT  (012) 

110  FORMAT  (64I1*2X*6I1) 

END 


APPENDIX  5  DATA  GENERATION  PROBLEM  2 

DIMENSION  M(64)*N(6) 

DO  10  I»l*4 
10  N(I)«0 
L«2 

DO  60  I«l*1000 

READ  INPUT  TAPE  6*100*4M( J) *^-1*64) 

IF  (M(l)>t>M<2)-t‘M(3)<*>M(4)‘*>M(5)>4((6)-M(7)-M(8)-M(9)-M(10))20*20*30 
20  N(5)«0 
GO  TO  40 
30  N(5)«l 

40  K«M ( 1 ) 4^M ( 2  )>M  ( 3 ) <^M(  4  )>M ( 9  )4M ( 6 ) 

CLA  K 
TZE*50 
SUB  L 
T2E*50 
SUB  L 
TZE»50 
SUB  L 
TZE*50 
N(6)«l 
GO  TO  60 
90  N(6)«0 

60  WRITE  OUTPUT  TAPE  7*100*<M( J) •Jb1*64) *  I N( J) • J-l *6) 

END  FILE  7 
CALL  RETURN 
STOP 

100  FORMAT  (64I1*2X*6I1) 

END 
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